Several classes of learning systems are known in the art. One broad class makes use of neural networks, either in the form of actual networks of electronic circuits or computer simulations of such networks. Learning in neural networks takes place by applying input signals to the network and adjusting connecting weights among neurons so that output signals from the network reflect the desired response to the input signals. Another broad class makes use of decision trees induced from training examples and the desired responses. Such decision trees can be reduced to sets of rules that can be applied to actual data.
Learning systems are becoming widely used as expert systems that acquire knowledge in a specific field or about a specific kind of problem. The input to expert systems can come from examples of input data and known responses to such data or from the knowledge of human experts. While learning systems can learn from examples alone, such systems can be made more efficient and accurate if background knowledge can be conveniently expressed as an input.
The results obtained by a learning system, whether in the form of weights for a neural network, rules from a decision tree or some other form, can be said to be a hypothesis or theory of the concept to be learned. The "target theory" for a learning system is the ideal theory that would always generate the correct output for a given set of input data. In practice, given data from the real world, it may never be possible to generate a target theory that always gives a "correct" result. However, in testing a learning system, it is often useful to determine how close the system can come to generating a known target theory from a set of training examples that correspond to the theory.
Some learning systems generate output in the form of a set of rules or "clauses," rather than a neural network or a decision tree. Such a set of clauses forms a logical theory; if learning is successful, then this theory is close to or similar to the target theory. An example of a prior-art learning system that generates logic clauses is FOIL, described in J. R. Quinlan "Learning Logical Definitions from Relations," Machine Learning, Vol. 5, No. 3, 1990.
A variety of techniques are known for taking advantage of special types of background knowledge, including constraints on how predicates can be used, programming cliches, overgeneral theories, incomplete theories and theories syntactically close to the target theory.
A way of extending FOIL to obey constraints on how predicates can be used is described in M. Pazzani and D. Kibler "The Utility of Knowledge in Inductive Learning," Technical Report 90-18, University of California/Irvine, 1990. An extension to FOIL that takes advantage of programming constructs or "cliches" useful in learning systems is described in G. Silverstein and M. Pazzani "Relational Cliches: Constraining Constructive Induction During Relational Learning," Proceedings of the Eighth International Workshop of Machine Learning, Ithaca, N.Y., 1991, Morgan Kaufmann. In some circumstances it is helpful to have a theory defining a concept that is related in some specific way to the target theory. A technique that uses overgeneral theories related to the target theory is IOE described in N. Flann and T. Dietterich "A Study of Explanation-Based Methods for Inductive Learning," Machine Learning, Vol. 4, No. 2, 1989. A way of incorporating incomplete theories is described in B. Whitehall and S. Lu "A Study of How Domain Knowledge Improves Knowledge-Based Learning Systems," Proceedings of the Eighth International Workshop of Machine Learning, Ithaca, N.Y., 1991, Morgan Kaufmann. Examples of incorporating syntactically close theories are shown in the Pazzani and Kibler article referred to above.
It is known how to integrate several different techniques for using background knowledge into the same learning system. An example is FOCL, described in M. Pazzani, G. Brunk and G. Silverstein "A Knowledge-Intensive Approach to Learning Relational Concepts," Proceedings of the Eighth International Workshop of Machine Learning, Ithaca, N.Y., 1991, Morgan Kaufmann. The problem, however, is to find a single technique for incorporating all these types of background knowledge, as well as other types of information about the target concept, into a learning system.